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ABSTRACT 

The purpose of this investigation was to develop a 
general framework for qualitatively analyzing the 1992 National 
Assessment of Educational Progress (NAEP) extended 

cons t ructed-response questions. The framework dimensions were based 
on information about the NAEP extended questions and linked to 
important ideas in mathematics education and cognitive psychology. A 
set of student responses (n=25) to an extended constructed-response 
question from the grade-4 assessment was analyzed qualitatively 
according to appropriate framework dimensions. The findings suggest 
that the student responses could be analyzed qualitatively, but 
further investigation is needed to verify the adequacy of the 
framework. Contains 12 references. (Author/MKR) 
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A FRAMEWORK FOR THE QUALITATIVE ANALYSIS OF STUDENT 
RESPONSES TO THE EXTENDED CONSTRUCTED-RESPONSE 
QUESTIONS FROM THE 1992 NAEP IN MATHEMATICS 

Patricia Ann Kenney, University of Pittsburgh (PA) 

The purpose of this investigation was ;o develop a general framework for analyzing the 
NAEP extended constructed-response questions qualitatively. The framework dimensions 
were based on information about the NAEP extended questions, and linked to important 
ideas in mathematics education and cognitive psychology, A set of student responses to an 
extended constructed-response question from the grade-4 assessment was analyzed qualita- 
tively according to appropriate framework dimensions. The findings suggest that the stu- 
dent responses could be analyzed qualitatively, but further investigation is needed to verify 
the adequacy of the framework. 



The National Assessment of Educational Progress (NAEP) is a Congression- 
ally-mandated survey of the educational achievement of American students and 
changes in that achievement over time. Since 1978. NA^P has assessed student 
performance in mathematics, with the most recent assessment conducted in spring 
of 1992. Compared to earlier NAEP mathematics assessments, the 1992 assess- 
ment was different in some important ways including closer alignment to the vi- 
sion for school mathematics as presented in the Curriculum and Evaluation Stan- 
dards for School Mathematics of the National Council of Teachers of Mathematics 
(1989). An innovative feature of the 1992 NAEP mathematics assessment in- 
volved the introduction of a new item type called the extended constructed-re- 
sponse question. As opposed to multiple-choice questions (which require students 
to select the correct answer from a provided set of answers) and regular-constructed 
response questions (which require students to generate their own numerical an- 
swer or to provide a very short explanation), extended constructed-response ques- 
tions not only require students to generate their own answers but also to express 
their mathematical ideas in writing and to demonstrate their depth of understand- 
ing. 

The 1992 NAEP grade-4 mathematics test included five extended constructed- 
response questions, and the grade 8 and grade 12 tests each included six such 
questions. Students were instructed to allow themselves five minutes or more to 
work on the questions. Instead of being scored "right or wrong," as were the 
multiple-choice and regular constructed-response questions, the extended ques- 
tions were evaluated according to a focused holistic scoring scheme with catego- 
ries ranging from "minimal" (score level 2) to "extended" (score level 5). Quanti- 
tative information about student performance on the extended tasks used on the 
1992 NAEP mathematics assessment formed the basis for a report by Dossey, 
Mullis and Jones (1 993). Acritical review (Silver & Kenney, 1993) of the Dossey, 
Mullis and Jones report noted that although the quantitative summary format was 



1 wish to thank Dr. Jinfa Cai and Dr. Edward A. Silver for their helpful comments on previous drafts 
of this paper. 



informative, the usefulness of the information was somewhat limited because no 
effort was made to analyze student responses with respect to the kinds of strategies 
and representations most frequently employed by students or with respect to the 
kinds of errors commonly made by students. 

The purpose of the investigation described in this paper was to develop a 
general framework for analyzing the NAEP extended constructed-response ques- 
tions qualitatively. The utility of the framework was then examined by conducting 
a preliminary qualitative analysis of student responses to a selected NAEP ex- 
tended task administered to fourth-grade students. 

Developing the Framework 

Initially, it was deemed beneficial to inquire whether the NAEPextended con- 
structed-response questions were developed with the idea that student responses 
would be analyzed qualitatively as well as quantitatively. Informal discussions 
with mathematics education professionals and test developers involved with the 
1992 NAEP mathematics assessment revealed that the extended questions were 
not developed specifically to be analyzed qualitatively and that no qualitative ana- 
lytic framework was ever developed. However, there was agreement among those 
most deeply involved with NAEP that a qualitative analysis of student responses 
to the extended constructed-response questions would be very beneficial, espe- 
cially to classroom mathematics teachers, mathematics teacher educators, and cur- 
riculum developers. 

Since no qualitative framework existed for the NAEP extended constructed- 
response questions, it was important to find sources of general information about 
these questions and the expected kinds of student responses to be evaluated ac- 
cording to a focused-holistic scheme, with the expectation being that this general 
information about the extended questions might suggest appropriate framework 
dimensions applicable at least to some (perhaps all) extended constructed-response 
questions. After existing information sources about the NAEP extended questions 
were consulted, it was determined that the general scoring guide for the NAEP 
extended constructed-response questions (called the "generic levels of performance" 
in Dossey et al., p. 89) provided the most useful information about possible di- 
mensions for the qualitative framework. In particular, the general scoring guide 
recommends that student responses be evaluated according to important criteria 
such as conceptual understanding (e.g., "Response contains evidence of concep- 
tual understanding" [quotations taken directly from the general scoring guide]), 
solution strategies, (e.g., "Methods of solution are appropriate and fully devel- 
oped"), error patterns (e.g., "Response contains major mathematical errors"), evi- 
dence of reasoning (e.g., "Response is logically sound"), and justification of an- 
swer (e.g., [through the use of examples] - Examples provided are not fully devel- 
oped"). 

Those criteria of conceptual understanding, solution strategies, error patterns, 
evidence of reasoning and justification of answer are also among the criteria rec- 
ognized by cognitive psychologists (e.g., Glaser, Lesgold, & Lajoie, 1 985; Royer, 
Ciscero, & Carlo, 1993) and mathematics education researchers (e.g., Charles & 



Silver, 1989) as important dimensions for measuring students' high-level perfor- 
mance. The evaluation criteria in the NAEP general scoring guide are also remi- 
niscent of the categories used in the QUASAR 1 project's qualitative analytic com- 
ponent. In QUASAR, a qualitative analytic framework has been used to report 
results of complex performance on open-ended, paper-and pencil tasks (which are 
similar to the NAEP expended constructed-response questions) with respect to di- 
mensions such as solution strategies, mathematical misconceptions, mathematical 
justification, and modes of representation (Cai, Magone, Wang, & Lane, 1995; 
Magone, Cai, Silver, & Wang, 1993). 

Based on information from the NAEP general scoring guide, important ideas 
from cognitive psychology, and the QUASAR qualitative analytic model, it was 
decided to select the following criteria as dimensions for the qualitative frame- 
work for the NAEP extended constructed-response questions: (a) conceptual un- 
derstanding; (b) solution strategies or modes of representation; (c) mathematical 
errors or misconceptions; and (d) evidence of reasoning. It is worth noting here 
that it was not the expectation that every NAEP extended question be evaluated 
according to all four dimensions. Due to differences in problem situations and 
content, not all of the dimensions are equally appropriate for every extended ques- 
tion. 

Using the Framework 

The dimensions of the qualitative framework were used to analyze student 
responses to a selected NAEP extended constructed-response task. The qualita- 
tive analysis itself was structured according to the model developed for the QUA- 
SAR project (Magone, Wang, Cai, & Lane, 1993): 1) conduct a logical analysis 
of the question to identify its cognitive requirements and content; 2) select appro- 
priate framework dimensions based on the results of the logical analysis; 3) apply 
the selected dimensions to a sample of student responses; 4) expand and modify 
the selected set of framework dimensions based on results from the sample of 
students responses; and 5) conduct the qualitative analysis using the final set of 
dimensions for analyzing student responses. The following sections of this paper 
focus on Steps 1-3 for qualitatively analyzing student responses to a 1992 NAEP 
grade-4 extended-constructed response question, hereafter referred to as "Pizza 
Comparison" and shown in Figure 1 . 



1 QUASAR (Quantitative Understanding: Amplifying Student Achievement and Reasoning) 
is a Ford Foundation sponsored project designed to enhance mathematics instruction in 
middle schools with high percentages of students from economically disadvantaged 
communities. One aspect of the project has been the development of the QUASAR Cognitive 
Assessment Instrument [QCAI]), a performance assessment used to measure the impact of 
these enhanced instructional programs on students* mathematical reasoning, problem solving 
and communication. Information about the QCAI can be found in Lane (1993) Silver and 
Lane (1993). 
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Sosi ate 1/2 of a pizza. 
Ella ate 1/2 of another pizza. 

Josd said that he ate more pizza than Ella, but Ella said that they both ate the 
same amount. Use words and pictures to show that Jos6 could be right. 

Figure 1. NAEP grade 4 extended constructed-response question: Pizza Comparison 

Step 1 : Logical Analysis of the Pizza Comparison Question 

The Pizza Comparison question was designed to "assess how well students 
are making the transition from whole number reasoning into using concepts asso- 
ciated with fractions" (Dossey et al., 1993, p. 91). Using a real-life setting of 
comparing quantities of pizza, the question measures students' understanding of 
the importance of the relative size of the object or unit in interpreting a fraction 
and taps into their knowledge of proportional reasoning. Concepts such as the 
importance of the size of the unit in fractions have been identified by mathematics 
education researchers (e.g., Behr, Harel, Post, & Lesh, 1992) as critical to the 
acquisition of a deep understanding of rational numbers. 

Step 2: Selected Appropriate Framework Dimensions 

Findings from the logical analysis of the Pizza Comparison question suggest 
the following as appropriate dimensions from the framework: 

• Conceptual understanding: understanding of the effect of relative 
difference in the size of the unit ("whole pizza"). 

• Modes of representation: use of pictures only, words only, or a com- 
bination of pictures and words. 

• Mathematical misconception: "1/2 is always 1/2." 

Step 3: Preliminary Results Using a Sample of Student Responses 

At the time this paper was written, the researcher had access to a small set of 
student response (n = 25) to the Pizza Comparison question. The preliminary 
findings from the qualitative analysis follow. 

Conceptual understanding* Over half of the student responses (n = 13) 
showed evidence of conceptual understanding of the importance of the relative 
size of the unit in comparison of fractions. The most common method of demon- 
strating the importance of relative size involved drawing two pizzas, one smaller 
than the other, dividing each pizza approximately in half, and labeling the larger 
one "Jos6" and the smaller one "Ella." In a few cases, students supplemented the 
labeled drawings of two different-sized pizzas with one-sentence explanations (e.g., 
"Jos6 could have had a bigger pizza than Ella."); some students even mentioned 



sizes commonly associated with commercially-made pizzas (e.g., "Jose ordered a 
large pizza and Ella ordered a medium pizza." "Her pizza was 8 inches, Jos6 could 
of had a 12 inch pizza"). 

Modes of representation. Most responses in the set involved a combination 
of words and pictures, a finding that is not surprising given these instructions to 
the student: "Use words and pictures to show that Jose could be right." Figure 2 
below shows typical examples of word/picture combinations. A few responses 
were expressed in words only (e.g., "Jos6's was large and Ella's was small [me- 
dium]."), while a few others consisted of unlabeled pictures. 



(a) - picture and complete sentence (b) - labeled picture 



Figure 2. Examples of word/picture combinations for the Pizza Comparison Question 

Mathematical misconceptions. Twelve responses were based on the mis- 
conception that "1/2 is always 1/2." In some responses, students drew two equal- 
sized pizzas, divided them in half, and wrote a comment such as "Jos6 ate his half 
and Ella ate her half; they both had 1/2 and they both ate the same amount." How- 
ever, other responses associated with the misconception that 1/2 is always equal to 
1/2 were based on drawing one pizza, dividing it in half, and designating the halves 
as "Jos6's" and "Ella's". This last example illustrates an error in understanding 
that was not anticipated in the logical analysis. The sentence, "Ella ate 1/2 of 
another pizza" (emphasis added) was a clue that the problem involved two differ- 
ent pizzas. However, some fourth-grade students most likely misunderstood or 
misread the problem. 



This study focused on the development of a framework for analyzing student 
responses to the extended constructed-response questions from the 1992 NAEP 
mathematics assessment. Results from a preliminary analysis of a small set of 
responses to one extended question suggest that the student responses could be 
analyzed according to the framework dimension, but that further study of the ad- 
equacy of the framework is needed using more responses and using all NAEP 
extended questions. 




Conclusion 
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